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ABSTRACT: Designing, validating, and deploying learning analytics tools for instructors or 
students is a challenge that requires techniques and methods from different disciplines, such as 
software engineering, human-computer interaction, computer graphics, educational design, and 
psychology. Whilst each has established its own design methodologies, we now need 
frameworks that meet the specific demands of the cross-disciplinary space defined by learning 
analytics are needed. In particular, LAK needs a systematic workflow for creating tools that truly 
underpin the learning experience. In this paper, we present a set of guiding principles and 
recommendations derived from the LATUX workflow. This is a five-stage workflow to design, 
validate, and deploy awareness interfaces in technology-enabled learning environment. LATUX is 
based on well-established design processes for creating, testing, and re-designing user interfaces. 

We extend existing approaches by integrating the pedagogical requirements needed to guide the 
design of learning analytics visualizations that can inform pedagogical decisions or intervention 
strategies. We illustrate LATUX in a case study of a classroom with collaborative activities. Finally, 
the paper proposes a research agenda to support designers and implementers of learning 
analytics interfaces . 1 

Keywords: Design, HCI, groupware, visualizations, design, dashboard, awareness, software 
engineering 


1 An earlier, shorter version of this paper (Martinez-Maldonado et al., 2015) is the foundation for this article, which has been 
extended in light of feedback and insights from LAK '15. 
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1 INTRODUCTION 

The field of learning analytics (LA) has emerged in recent years as a multidisciplinary research area with 
the aim of improving the overall learning experience for instructors and students. Several authors have 
highlighted the potential impact of this discipline to enhance teaching practice and student learning 
(Ferguson, 2012b; Siemens, 2012; Siemens & Baker, 2012; Verbert et al., 2013) but at the same time, 
there is also a recognition of the importance of establishing a connection with other research fields such 
as the Learning Sciences (Siemens, 2012), Human-Computer Interaction (HCI) (Verbert et al., 2013), 
Information Visualization (Chatti, Dyckhoff, Schroeder, & Thus, 2012) and Computer Graphics (Duval, 

2011) . This is because the design of tools in this space demands challenging interdisciplinary processes 
(Balacheff & Lund, 2013) in order to satisfy multiple goals and stakeholder requirements and to leverage 
the knowledge base of these different disciplines. There is substantial research demonstrating the value 
of LA (Ferguson, 2012b; Siemens & Baker, 2012), describing the basic requirements for LA tools 
(Dyckhoff, Zielke, Bultmann, Chatti, & Schroeder, 2012), envisioning the future of this field (Siemens, 

2012) , and providing general analytical solutions for specific contexts (Dyckhoff et al., 2012). However, 
little attention has been paid to exploring frameworks that can systematically help designers to develop, 
evaluate, and deploy effective LA tools. This is a critical time for learning analytics, because there is clear 
recognition of the need for effective interfaces but there are no standards yet. Our approach aims to 
facilitate the exploration of potential LA tools and to speed up the creation of validated tools for various 
purposes. 

The area of Software Engineering (SE) provides established methodologies to help designers recognize 
processes, understand the requirements for interactive systems and iteratively evaluate these as they 
are built (Sommerville, 2011). The area of visualization and computer graphics also has established 
guidelines for designing general visual analytics interfaces (Thomas & Cook, 2006). Similarly, the HCI and 
User Experience (UX) disciplines have a large array of methods for establishing user needs, and 
designing and evaluating interfaces (Hartson & Pyla, 2012). However, these methodologies alone are not 
sufficient for building LA tools because they do not take account of the learning context. Dillenbourg et 
al. (2011) stated that traditional usability testing (mostly focused on individual or group interactions 
with computers) is not enough to prepare designs for the complexity of a learning context. The same 
authors suggested considering "usability at the classroom level." For example, in a classroom, instructors 
typically have to deal with multiple constraints and contingencies such as time limitations, curriculum 
alignment, lesson plans, physical space constraints, unexpected events, and tool usage. This complexity 
also occurs in blended and online learning settings (Prieto, Dlab, Gutierrez, Abdulwahed, & Balid, 2011) 
where LA tools are commonly used and potentially have a particularly valuable role. Similarly, Siemens 
(2012) proposed a holistic view of LA that includes such practical issues, but also aspects related to the 
data, such as openness, accessibility, and ethics, as well as the particular pedagogical goals and usage 
context of the LA tools (e.g., detecting at-risk students, supporting self-awareness, or enhancing 
instructor awareness). 
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These complexities point to the need for new design methodologies for LA tools, providing a 
pedagogical underpinning and considering the different actors (e.g., instructors and students), the 
dimensions of usability in learning contexts (Dillenbourg et al., 2011) (individuals, groups of students, 
and the classroom), the learning goals, data sources, and the tasks to be accomplished. This paper 
presents LATUX (Learning Awareness Tools — User experience), a workflow for designing and deploying 
awareness tools for technology-enabled learning settings. By awareness tools, we refer to those that 
provide a user (instructor) with an enhanced level of awareness of what other actors (students) are 
doing in the learning space. The workflow extends well-established design approaches for developing 
user interfaces (Uls) by integrating the pedagogical requirements, collecting the relevant data from 
students, and evaluating the awareness tools. The evidence supporting the design of the LATUX 
workflow is illustrated with a case study of the iterative development of visualizations for in-classroom 
technology-enabled group activities to assist instructors in monitoring student collaboration and 
progress (Martinez-Maldonado, 2014). 

The rest of the paper is organized as follows: The next section provides an overview of the current 
related LA methods followed by a brief description of the most relevant methods for interface 
development and evaluation used in SE and HCI. Then, we define the LATUX workflow, with its four 
principles and five stages. The workflow is illustrated in the final section in terms of the iterations of 
design, evaluation, and the deployment of visualizations of student group activity in our case study. 

2 RELATED WORK 

2.1 Awareness Tools: Visual Analytics, Learner Models and Dashboards 

There has been growing interest in the use of visual representations and interaction techniques that 
exploit the human perception to let instructors or institutional decision makers see, explore, and 
understand a varied range of sources of student data simultaneously (Duval et al., 2015; Kay & Bull, 
2015; Ritsos & Roberts, 2014). This area has been called Visual Learning Analytics, making allusion to the 
more general field of Visual Analytics (Thomas & Cook, 2006) that focuses on the provision of 
interactive, visual support for assessment, planning and decision making in general. Nevertheless, for 
the case of learning and teaching, a number of visual tools have been targeted to help teachers, 
learners, and other stakeholders gain awareness of, analyze, or reflect on relevant traces of the learning 
activity collected from the various learning spaces where students interact. These tools include, for 
example, Open Learner Models (OLMs), mostly targeted to students (Bull & Kay, 2007), Academic 
Analytics, mainly focused at the institutional level (Campbell, DeBlois, & Oblinger, 2007), scrutable group 
models (Upton & Kay, 2009), or teacher's dashboards (Verbert et al., 2013). We can consider that these 
visual tools used in learning environments are different forms of learning analytics interfaces. 

The need for a framework to analyze and identify effective visualizations is one of the key aspects that 
must be addressed to advance the field of learning analytics (Duval et al., 2015). The main issue is how 
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to identify the real value of visualizations when used in real learning contexts, and the impact on the 
learning experience. The trade-off to consider is between the relevance of the data that might be 
presented and the manageable level of complexity for the stakeholders, in the context where they make 
use of the LA tool. For example, in visualizations provided to instructors to estimate academic risk, 
Ochoa (2015) identified the need to include a sense of the accuracy or source of uncertainty. These 
visualizations provide instructors with a nuanced description of the underlying results produced by the 
algorithms. What is the actual impact of including this information in a visualization? How are the 
instructors affected? These questions can be applied generically as in other visualizations, such as 
networks to represent short answers (Bakharia & Dawson, 2015), visualization of tags previously 
collected by students in a learning environment (Niemann et al., 2015), networks of aggregated 
concepts derived from collaborative writing (Hecking & Hoppe, 2015), or interactions captured in an 
interactive surface (Charleer, Klerkx, & Duval, 2015). 

The main challenge for creating such a framework is to make it generic enough to apply to the wide 
variety of possible visualizations, but still provide meaningful insight into the design process. The 
consequence of the lack of such formalism translates into a proliferation of visualizations exploring the 
limitations of what can be shown, instead of tackling the much-needed goals of relevance, usability, user 
satisfaction, and overall effectiveness. This paper aims to address this by providing a generic workflow 
targeted to support the design, validation and deployment of learning analytics interfaces. 

2.2 Learning Analytics Methods for Pedagogical Intervention 

Technology-enabled learning environments, whether online or face-to-face in the classroom, have the 
advantage of capturing copious amounts of data left in students' digital footprints (Duval, 2011; Greller 
& Drachsler, 2012). When this data is made accessible to instructors, it can provide valuable evidence- 
based insights into student learning. Using data-mining techniques, the data captured from student 
interactions can be aggregated and analyzed by instructors to make better-informed pedagogical 
decisions. For example, data on student use of a Learning Management System (LMS) can reveal 
patterns of student engagement with certain tools (Beer, Jones, & Clark, 2012; Macfadyen & Dawson, 
2010). However, these patterns are best understood by instructors when made available and visualized 
through user-friendly and intuitive LA tools. The challenge is to offer a low effort start-up for non¬ 
technical users while offering additional functionality for more advanced users (Siemens, 2012). 

The target audience for LA tools must be considered during the design process to ensure the insights 
gleaned from the data become actionable (Olmos & Corrin, 2012). In the case of course-level analytics, 
visualization tools should highlight the intended learning goals identified by the instructor and whether 
these have been achieved (Gluga et al., 2013), or even allow instructors to apply the performance 
indicators appropriate for their learning context (Dyckhoff et al., 2012). The ability to observe whether 
students are engaging with a learning tool or progressing as intended allows instructors to identify when 
to intervene in order to support the students. This is especially challenging in online learning 
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environments (Dyckhoff et al., 2012) and in large classes. For example, Social Networks Adapting 
Pedagogical Practice (SNAPP) represents student interactions within an online discussion forum to help 
instructors identify when to consult with students who are not engaging or when to change the nature 
of the activity (Lockyer, Heathcote, & Dawson, 2013). Other learning analytics dashboards, such as 
Purdue University's Course Signals, provide data visualizations (traffic signals) to monitor performance 
and trigger early intervention when students may be at risk of poor performance or dropping out of a 
course (Arnold & Pistilli, 2012). While there is no shortage of evidence to support the argument that LA 
tools to visualize student activity at the course-level are needed to inform instructors about their 
students' learning behaviour, there is limited research on effective workflows for designing such tools, 
particularly when focusing on data collected from face-to-face learning environments. This paper 
addresses this gap by proposing a design workflow that combines the rigour of HCI paradigms regarding 
user experience with a solid pedagogical underpinning. 

2.3 Designing User Interfaces 

Several methodologies have evolved over many years, to facilitate the process of software development 
and, specifically, user interface design. From a general SE perspective, these methodologies are 
commonly known as lifecycles (Sommerville, 2011) which, for example, may include waterfall, spiral 
development, prototyping, and extreme programming approaches. The purpose of the lifecycles in HCI 
is to help designers develop more efficiently and, for the case of user interfaces, to enhance the user 
experience. In other words, the lifecycles provide best practices for developers and designers. 

The lifecycles in both SE and HCI disciplines have similar structures and activities. These include 
requirement identification, design, and evaluation. However, the methodologies for designing user 
interfaces in SE and HCI have been formulated almost independently (Hartson & Pyla, 2012). There are a 
number of different HCI lifecycles, such as Mayhew's usability lifecycle, the Star lifecycle, the LUCID 
framework of interaction design, and the Wheel model (Hartson & Pyla, 2012). The latter is a more 
recent elaboration of the iterative process model for usability engineering that provides a general 
framework into which designers can fit existing or new techniques or methods to apply "best user 
experience practices" (Helms, Arthur, Hix, & Hartson, 2006). This is particularly relevant as a foundation 
for our LATUX workflow. Broadly, most of the lifecycles above have four generic stages (analysis, design, 
implementation, and evaluation) that should be performed iteratively, with potential for each stage to 
feed back to refine an earlier one. This iterative process for interface design can lead to a higher quality 
user experience. 

Adding to the challenge for LA, Bevan (2009) argued for greater use of usability standards, but 
acknowledges that generic standards may fail to meet all the specialized needs of designers in specific 
areas (e.g., for LA tools design). A widely accepted standard relevant for developing interfaces for LA is 
the "Human-centred design for interactive systems" standard — ISO 9241-210 (2009). Its six principles 
are as follows: 1) the design should be based upon an explicit understanding of users, tasks, and 
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environments; 2) users should be involved throughout the design; 3) the design should be driven by 
user-centred evaluation; 4) the process is iterative; 5) the design should address the whole user 
experience; and 6) the design team should be multidisciplinary in terms of skills and perspectives. 

The LATUX workflow is based on this standard, and the evaluation-centred UX model (the Wheel) 
lifecycle concept (Hartson & Pyla, 2012). Its novelty resides in the fact that it instantiates the user 
experience lifecycle with specific evaluation stages and requirements for designing awareness tools 
aimed at improving the learning experience, using data captured from students' activity with learning 
technologies. 

2 THE LATUX WORKFLOW 

The LATUX workflow is defined by 1) the principles on which it is grounded, and 2) the evaluation- 
oriented stages that compose it (see Figure 1). The entire workflow is evaluation-oriented. Hence, each 
stage evaluates different parts of an awareness tool. The workflow has one problem identification stage 
and four evaluation stages. These principles and the stages are described in the rest of this section. 

2.1 LATUX Principles 

Based on LA literature and our first-hand experience, we identified a set of principles to take the 
workflow beyond generic development lifecycles. These principles are: 

Principle 1: The development process in LA has a strong research component. In traditional software 
design, development is commonly based on iterative requirements elicitation. The classic UX approach 
involves exploring the design space and gaining feedback iteratively, with careful evaluation on each 
iteration. By contrast, developing effective awareness tools for learning settings also requires research 
to explore the new possibilities that learner data can offer to support instructors. As with every field 
that has been revolutionized by technology, LA tools can have a transformative role in education, in 
addition to improving its processes. Research needs to be carried out concurrently to understand how 
these changes can take place. At the current state-of-the-art, there is a prominent gap between 
research and practice in LA (Siemens, 2012) indicating the importance of this principle in the proposed 
workflow. This opportunity to close the gap is what Duval (2011) has highlighted as "Data-based 
Research on Learning." 

Principle 2: The impact of authenticity; where the term authentic captures the nature of existing learning 
contexts without LA tools. In line with recognized UX practice, we need to distill the nature of learning 
contexts and how they impact the needs of each stakeholder and act as drivers for interface design. This 
aspect highlights the need to go beyond standard usability. Testing with a small number of users is a 
general method to evaluate interfaces and user experience. However, for awareness tools, the 
contingencies of real learning scenarios may affect and be crucial for the adoption of technology 
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(Dillenbourg et al., 2011). As also noted by Rodriguez Triana, Martinez Mones, Asensio Perez, & 
Dimitriadis (2014), we need to align the design of the learning activities (tools usage, tasks, pedagogical 
strategies) and the learning analytics solutions to support instructors more effectively. In our workflow, 
each evaluation stage aims to increase the level of match between the awareness tool and the demands 
of authentic learning contexts, while moving towards the deployment of the tool in-the-wild (the 
classroom). 

Principle 3: Holistic design and evaluation. In the same way that the LA field is multidisciplinary and 
focused on understanding systems as a whole (Siemens & Baker, 2012), the design process to build LA 
tools must consider the complexity of learning environments. This includes the features of the learning 
tools, the adopted pedagogical approach, the physical space, the student social roles, etc. Additionally, 
the evaluation of the tools should also be holistic. For example, rather than evaluating the effectiveness 
of just the user interface of the awareness tool itself (in isolation), the evaluation should consider the 
tool in the context of the learning setting and its further impact on the actual learning-related activities 
— e.g., student retention, change in learning behaviour (Verbert et al., 2013), or instructor's attention 
(Martinez-Maldonado, 2014). This holistic view translates into a close relationship with Learning 
Sciences (Siemens & Baker, 2012). 

Principle 4: Evaluation is based on data and its meaningfulness. In addition to the evaluation of the Ul, 
the design of LA tools has an additional dimension for evaluation; i.e., making sense of the student 
activity data or interpreting it in order to inform pedagogical decisions. A crucial part of the design and 
evaluation process includes the ways to distill data, present it, and validate it according to the 
pedagogical elements of learning and instruction (Knight, Buckingham Shum, & Littleton, 2013) including 
representation, purpose, timing, context, learning goals, teaching goals, epistemic beliefs, and 
assessment approaches. For example, a specific data visualization may enlighten instructors when they 
reflect on student progress over the semester, but the same visualization may be useless in the 
classroom, or in other learning contexts. Thus, the design process should not only deal with questions 
about what data to present and how that data should be presented, but also where and how it will be 
used. While the evaluation in terms of usability and user experience still applies for LA tools, the 
workflow presented in this paper is not a software development or user experience lifecycle. This 
process incorporates and draws on the body of knowledge about learning contexts such as the learning 
cycle, where the students' learning activity or instruction is the focus of attention. Further, the design 
and evaluation of LA tools should be connected to both the context of the learning activity and the 
context of usage. 

2.2 LATUX Stages 

Figure 1 shows the conceptual representation of the stages featured in our proposed iterative workflow 
for designing and deploying awareness tools in the classroom. The problem definition is the first stage of 
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recommended practice in HCI and SE. The figure also shows the four evaluation-oriented stages that 
constitute the iterations towards deployment. The five stages are described below. 

2.2.1 Stage 1 — Problem identification 

Most literature presented above, concerning LA, has explored the different elements of the problem 
definition (Siemens, 2012; Siemens & Baker, 2012; Verbert et al., 2013). The first stage aims to identify 
the requirements for LA and user interface design. This process offers an opportunity to identify possible 
new and radical features that can be offered by the data to address stakeholder needs, but where the 
stakeholders may not realize this. As noted under the first stage, Problem identification, in Figure 1, the 
first two questions to consider when identifying the LA-related problem concern the requirements and 
the unexplored possibilities that could be investigated. This can also lead to the consideration of 
multidisciplinary knowledge or expertise needed for the development of the learning analytics solution. 
For example, basic knowledge about information visualization is key when designing a dashboard; 
foundations of HCI can also be required to communicate learner's captured interactions to the 
instructors. 

The third question concerns the identification of the stakeholders. Although LA has mostly focused on 
informing and empowering instructors and learners (Siemens & Baker, 2012), Siemens (2012) identified 
other relevant stakeholders including faculties, departments, the whole institution, researchers, 
corporations, and government funders. From a SE perspective, stakeholder identification is a basic step 
for successful design (Sommerville, 2011). Hence, the design of LA tools should adopt this, carefully 
identifying the target stakeholders. 

Once the stakeholders have been identified, the available data sources will need to be considered. 
According to Verbert et al. (2013), to date, research on LA has mostly focused on using student data 
linked to resource usage, interaction time, number of artefacts created, collaborative interactions, and 
task results. By contrast, assessing the impact of the data displayed in awareness tools has not been 
deeply explored. In addition, there are many additional data sources that can offer new insights into 
student learning (Blikstein, 2013), but these have been under-explored in the LA field. They include 
gesture sensing, wearable biosensors, head trackers, infrared imaging, depth imaging, and eye tracking. 
Even manual tracking of activity has not been widely used in LA (Verbert et al., 2013). As a continuation 
of the data source identification, Verbert et al. (2013) also indicated that the design of LA tools should 
include an analysis of the feasibility and trade-offs of considering alternative data sensors used to 
capture more complete student data. Currently, most LA solutions are based on application logs and 
forms of audio and video capture. The problem definition should include the analysis of the implications 
of introducing additional sensors in terms of the potential undesirable impact on the learning 
environment. Other elements of the problem definition addressed by LA research, as discussed below, 
are the following: stakeholder identification, data sources, types of data logging, features of the learning 
setting and design for evaluation. 
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Alongside identification of the data sources, the learning and teaching approach or intended pedagogy 
should be considered. This is in line with the argument by Verbert et al. (2013) that there is a need for 
more research to identify the settings where particular LA approaches work well, and also to identify 
limitations and alternatives for learning environments where some LA solutions have been unsuccessful. 
This suggests a strong link between the development of LA tools and the context as a whole, and thus 
the risk of over-generalizing LA solutions. For example, the learning environment as well as the details of 
the learning context and tools deployed such as the activities and assessments, the student's prior 
experiences, technologies used, and intended learning outcomes should be considered in order to make 
sense of the data representing student activity and to make recommendations for other similar learning 
environments rather than over-generalizations. 


Another key aspect to consider in an early stage of the design of awareness tools is the definition of the 
learner's rights, the student's data ownership, and ethical issues that can emerge and which can be 
specifically relevant for learning analytics (Ferguson, 2012a; Siemens, 2012). Slade and Prinsloo (2013) 
stated that it is important to understand both the opportunities and ethical challenges of learning 
analytics, which are socially and epistemologically shaped by the educational context and purpose of the 
awareness tools. Institutions, researchers, designers and/or practitioners must consider privacy and 
legal issues when defining the protocols for collecting, analyzing, interpreting, and storing student data 
(Bienkowski, Feng, & Means, 2012). Some early actions can be taken to address possible ethical issues in 


□ What are the requirements? 

□ What are the (unexplored) 
possibilities? 

□ Who are the stakeholders? 

□ What are the available data sources ? 

□ What is the intended pedagogy? 

□ How can the data be captured? 

□ What is the learning context? 

□ What are the learning tools? 

□ How to evaluate the LA tools? 

□ What ethical issues can emerge? 

□ How is data to be collected, used, 
shared and stored? 



THE LATUX WORKFLOW 


1- Problem identification 


What visualisations can be 
generated in real-time? 

How can interactivity be 
supported? 

How can the impact of the tool 
be tested? 

How can contingencies be 
addressed? 

How can interventions and 
change of behaviour be tested? 


Example: Usage of the tool in a few authentic 
sessions _ 



What candidate visualisations should be 
generated? 

How to mock up/sketch visualisations? 
How would the visuals be generated? 
What data is required? 

How would the evaluation mimic real 


2- Low-fidelity prototyping 


Example: paper-based visualisations 


How can visualisations be 
dynamically generated? 

How can the data be 
generated/processed? 

What conditions of the setting can 
be simulated? 

How can the interventions be 
simulated? 

How can interaction be supported? 
How can the impact of the LA tools 
be evaluated? 


3- Higher fidelity prototyping 


Example: Simulated class 


□ What limitations are imposed by the in-the-wild conditions? 

□ What ethical issues can restrict the evaluation of the tools 
in-the-wild? 


Figure 1. Conceptual Representation of the LATUX Workflow for Designing and Deploying 
Awareness Tools. Each stage features the guiding questions that designers can consider before or 
while designing and validating learning analytics visualizations. 
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subsequent stages of the development. For example, policies can be defined for obtaining a student's 
consent to use their data (Slade & Prinsloo, 2013). It can also be considered to agree on a data privacy 
statement or disclaimer. The de-identification of student data can also be required to accomplish 
specific purposes (e.g., for sharing data with third-party entities or for conducting research). All these 
concerns must be addressed within the constraints of national and broader privacy legislation. This 
broadly considers that the learners should be able to access their data and be in control of its uses. But 
the understanding of appropriate use of learning data is still evolving. Broad principles for ethical 
management of learning data in MOOCs were established at the Asilomar Convention for Learning 
Research in Higher Education in 2012. 2 This aimed to facilitate progress in harnessing the potential 
power of the data, but at the same time respecting the learner's rights. 

Finally, as the last question under Problem identification in Figure 1 shows, it is important to define how 
to evaluate the effectiveness of the use of LA tools in the target learning setting. This means designing 
the evaluation of the user experience according to the principles this workflow is grounded upon (e.g., 
considering the authenticity of the learning setting, meaningfulness, ethical concerns that can emerge in 
studies in-the-wild, etc.). Different measurement techniques should be used, as perception may not be 
an accurate way to evaluate the impact of a tool (Hartson & Pyla, 2012). Measuring users' actions and 
the impact of such actions are also an indirect measure of the effectiveness of the learning analytics 
tool. These include usability testing, measuring the impact on teaching, learning gains, reduction in 
misconceptions, improved collaboration, enhanced instructor awareness, reduced load on the teacher, 
enhanced management of instructor time, etc. In addition, the conditions in which the evaluation will be 
performed (e.g., in the classroom, with simulated class conditions, simulated data, etc.) are required. 
Further details on this point will be discussed in the following stages of the LATUX workflow. 

2.2.2 Stage 2 — Low-fidelity prototyping 

The idea of prototyping in the early stages of development is to provide a quick and flexible preliminary 
vision of the proposed system (Hartson & Pyla, 2012). To accomplish this, a prototype must contain 
some key elements of the system without actually building the system. Low-fidelity prototypes are 
initial, high-level representations of the intended design, which may be sketched or be illustrated using 
graphical wireframe tools. They are appropriate when the details of the tool have not been defined or 
are likely to change, so the construction of prototypes should involve little effort. In this stage, we 
propose the construction of low-fidelity prototypes for the purposes of: 1) testing a wide range of 
possible visualizations that can be part of the intended awareness tool; 2) testing usability and data 
sense making in the early stages of the design; and 3) evaluating and triggering dialogue with 
stakeholders before actually implementing the LA tool. To achieve these, Figure 1 lists a set of questions 
that may guide a designer in deciding: 1) how to generate and evaluate low-fidelity prototypes of 
awareness tools including the types of visualizations to be generated; 2) the strategies to be used to 


2 http://asilomar-highered.info/ 
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initially illustrate and generate the visualizations; 3) the data required; and 4) how the evaluation 
strategy of the low-fidelity prototypes will strive to mimic the use of such visualizations in a real setting. 

In LA, low-fidelity prototyping may focus on either the evaluation of the Ul (user interface oriented 
prototype, e.g., a paper-based version of the interface where some visualizations can be drafted with 
simulated data) or the actual information conveyed in the visualizations (data-oriented prototypes, e.g., 
visualizations of data captured by the actual learning system or from shared datasets). The first case is 
closer to a typical HCI evaluation, while data-oriented prototyping is perhaps more relevant (and useful) 
in LA. We will provide an example of this in the Case Study described in the next section. 

In either case, low-fidelity prototypes are commonly paper-based representations of parts of a system 
or tool. They are low cost and are highly effective to inform the design compared with higher-fidelity 
versions of the tool (Tohidi, Buxton, Baecker, & Sellen, 2006) (see Table 1, column 1, first row). This is 
because one can readily create multiple prototypes, each exploring quite different approaches, and so 
gain insights into the relative strengths and weaknesses of each of them. This also enables participants 
in the evaluation studies to comment on them in comparative terms, without feeling the discomfort of 
being negative. They also allow evaluations with multiple users and under controlled lab conditions (e.g., 
it is easier to evaluate paper-based versions of a dashboard in one-hour interviews with multiple 
instructors than when the actual system is used). 

By contrast, low-fidelity prototypes in LA have many limitations (see Table 1, column 1, row 2). They 
cannot replicate the connection between what occurs in an environment and the generation of data at 
that precise instant. In addition, these evaluations completely ignore the effect of interventions (e.g., 
instructors' actions or provision of feedback) or changes in student behaviour. They also afford very 
limited understanding of the user experience under authentic learning conditions (see Table 1, column 
1, rows 3 and 4). An alternative way to get rapid feedback in the initial stages of design is through the 
use of Wizard of Oz studies (Dahlback, Jonsson, & Ahrenberg, 1993). In this method, an interface 
prototype looks as though it operates as intended, but actually lacks the backend to drive the full 
functionality and so a person must provide this. 

2.2.3 Stage 3 — Higher-fidelity prototyping 

The next stage in the workflow is to implement higher fidelity prototypes by creating at least a partial 
backend with meaningful data. From a HCI perspective, a high-fidelity prototype is a more detailed and 
realistic representation of the designed tool than a low-fidelity prototype (Hartson & Pyla, 2012). It may 
include details of either the appearance or the interaction behaviour needed to evaluate how users 
interact with the tool or the data. High-fidelity prototypes require more effort and time, but the 
evaluation is still less expensive; these prototypes are faster to produce and more flexible than 
developing the final tool. 
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For LA awareness tools specifically, we discuss the simulation and generation of student data and the 
conditions of the learning setting. This adds some degree of authenticity to the evaluation since the 
simulation may involve real student data and real-time usage. As indicated in Table 1 (first row, column 
2), using simulation prototypes allows for the inclusion of a time factor and, therefore, the evaluation of 
the decision making process on the fly (e.g., it is possible to analyze what an instructor would decide to 
do, right after looking at the tool). Similar to low-fidelity prototypes, evaluations with more users can be 
performed under lab conditions compared with more authentic but costly pilot studies. In addition, 
some aspects of interaction design can be evaluated (e.g., analyzing how users would explore the data). 

Experiments that are more complex can be designed to evaluate the user experience with a simulation 
tool since subjects can just focus on the details of the visualizations while the tasks can be replicated for 
multiple users in less time. However, as noted in Table 1 (second row, column 2), the impact of student 
or instructor actions are still ignored. Hence, the simulation of a tools' usage does not reflect what 
would actually happen in a real environment where students or the instructors can actively respond to 
the data presented via the awareness tool. Therefore, the effect of the tool on instructor or student 
actions cannot be tested. As a result, the designer should consider the trade-off between the low degree 
of authenticity with this type of prototyping and the effort needed to build it. Figure 1 presents some 
questions that can help the designer evaluate: 1) the ways visualizations can be dynamically generated; 
2) how data can be generated and processed; 3) the conditions of the learning environment that can be 
simulated to evaluate the use of the visualizations within the awareness tool as authentically as 
possible; 4) how interventions can be simulated (e.g., the teacher speaking with students who may not 
be engaging with the tool as intended); 5) how such interactions can be supported; and finally, 6) how 
the impact of the tool can be evaluated without actually measuring learning or behavioural change. 


Table 1. Strengths and limitations of the evaluation-oriented stages of the workflow. 


Low-fidelity 

prototyping 


Higher-fidelity 

prototyping 


Pilot studies 


Validation in-the-wild 


Strengths 


•Testing usability and 
sense making in early 
stages 

•Testing with multiple 

people 

•Very cheap 


•Simulated decision making 
on the fly 

•Testing with multiple 
people 

•Deeper evaluation by 
users 

•Interactivity can be tested 
•Cheap 


•Authentic -live usage of the tool 
•Interactivity can be tested 
•Interventions affect the use of the tool 
•Impact of the tool on users' can be tested 
•Contingencies considered 


•Cheaper than full deployment 


•Moregeneralisable 
•Testing with multiple people 
•Reduced 'novelty-effect' 


Limitations 


•Time factor missing 
•Interventions cannot 
be tested 

•Interactivity cannot be 
easily tested 


•Interventions cannot be 
tested 

•More expensive than 
static prototypes 


•More expensive than 
prototyping 

•Testing with fewer people 
•Less generalisable 


•The most expensive 
•Contingencies 
•Limited experimental 
conditions 


Controlled 

conditions 


Authenticity 
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2.2.4 Stage 4 — Pilot studies: Classroom use 

The fourth proposed stage before deploying an awareness tool in-the-wild is the evaluation of the user 
experience in pilot studies. A pilot study is commonly accomplished under conditions similar to a real 
deployment but on a smaller scale (e.g., in a limited number of classroom sessions or during a limited 
period in a course). A pilot study can help prove a concept and observe the live usage of the tool in an 
authentic context. This is recommended as part of the design workflow to minimize the risk of deploying 
an LA tool on a large scale while there is still a need for more research and evaluation. 

Key new knowledge can be gained from pilot studies including testing interactivity. This can include the 
analysis of the impact of the interventions and change of behaviours because of using the tool. In pilot 
studies, it is also possible to test the effect of the tool and the unexpected events that may occur on 
learning and instruction (see Table 1, column 3, first row). Figure 1 provides five questions that should 
be considered when designing a pilot study: 1) the types of visualizations that can be generated in real¬ 
time (e.g., to be used when the teacher wants to whether during the class or external); 2) how 
interactivity with the awareness tool would be supported during the pilot; 3) strategies for testing or 
evaluating the impact of the tool; 4) how contingencies will be addressed, and 5) how teacher 
interventions or changes in student behaviour will be tested or evaluated. 

However, pilot studies are typically more expensive in terms of effort and coordination, compared with 
evaluating low- or high-fidelity prototypes in the lab (Hartson & Pyla, 2012). Added to that, results 
obtained from pilot studies are not necessarily generalizable. Setting controlled variables can be 
restrictive; depending on how authentic the learning conditions are (e.g., pilot studies in real schools 
may impose limitations on the range of items that can be evaluated). In addition, the amount of 
evidence and subjects can be more limited than performing shorter but more numerous evaluations 
with prototypes under lab conditions (see limitations in Table 1, column 3, second row). 

2.2.4 Stage 5 — Validation in-the-wild: Classroom use 

Evaluating in-the-wild has become a standard method within HCI testing (Brown, Reeves, & Sherwood, 
2011). In-the-wild testing consists of field trials of experimental systems with groups of subjects in 
relatively unconstrained settings outside of the laboratory, at a larger scale and for a longer duration 
than in pilot studies. For LA research and practice, this can help designers recognize the actual 
deployments as another evaluation stage of user experience. Similar to a pilot study, deployments in- 
the-wild provide an authentic test-bed to analyze not only usability and user experience, but also the 
usage and the impact of awareness tools in an unpredictable learning environment and within the 
context of the pedagogical requirements and goals (see Table 1, column 3, first row). 

The actual use of the awareness tool can be tested, considering the possible contingencies that may 
occur. As the authenticity of the setting is higher and the duration of the study can be longer than a pilot 
study, the results of the evaluation of the awareness tool may be more generalizable to real world 
contexts and use. In addition, the longer-term usage of an awareness tool can minimize the "novelty- 
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effect" inherent in introducing new technology. This effect cannot be easily analyzed in smaller pilot 
studies or in prototype testing. This aspect is important for designing and testing LA tools since the 
ultimate goal of a successful deployment is the sustained usage of the tool for its intended learning or 
teaching goal. 

In addition, testing the tool in-the-wild may require more challenging learning management, ethical and 
privacy issues to be addressed given the larger scale use of the tools compared with pilot studies (Table 
1, column 3, second row). These aspects may influence the design of an awareness tool and inform the 
two key questions the designer should consider prior to deploying an awareness tool as noted in Figure 
1. Namely, the designer should consider the limitations posed by "in-the-wild" conditions (e.g., class size 
producing a larger data set, technological infrastructure or lack thereof, and the extent of teachers' data 
literacy to be able to make sense of the visualizations represented) and any particular ethical issues 
emerging from the evaluation of the tool in-the-wild. We include this consideration at this stage, in 
addition to the first stage's planning, because the earlier stages work towards a tool that can be used in 
multiple classes of envisaged settings. When it comes to testing in a particular class and context, the 
pragmatics of each will need to be considered. 

Finally, evaluating LA tools in authentic deployments is the most expensive stage in a design workflow. 
They may also be more restricted to fit the context. Testing under authentic conditions can also affect 
the use of the tool and the evaluation. Typically, they offer less flexibility and this means that one 
cannot ensure the systematic testing of a broad coverage of diverse cases. By contrast, a lab setting 
makes it possible to ask the users to work through a carefully designed and comprehensive set of 
artificial tasks (Table 1, column 4, second row). 

2.2.5 Iterative Design-Deployment-Evaluation 

Finally, bridging the gap between practice and research calls for communication of what happens in the 
deployment to inform the possible tuning or re-design of the tools. This invites the definition of 
iterations of design either to develop the LA tools in small incremental steps, or to improve certain 
elements of the Ul or data processing. Mendiburo, Sulcer, and Hasselbring (2014) highlights the 
importance of iterative interaction design to generate their LA tool, but other than this example, 
iterative design in LA has not been explored and reported in depth. The designer may want to respond 
to some questions once the system, or part of it, has already been deployed, such as what evaluation 
stages may be needed in an iteration (e.g., some functionality of the awareness tool may not require all 
stages of prototype testing or pilot studies)? What is the goal of the current iteration? What knowledge 
should be gained from a research perspective? How many iterations are required? As shown in Table 1 
(rows 3 and 4), each stage within an iteration of the proposed workflow features a trade-off between 
the authenticity and degree of control of the learning setting conditions. 
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3 CASE STUDY 

We illustrate our workflow by describing the iterative development of the MTDashboard (Martinez- 
Maldonado, Dimitriadis, Kay, Yacef, & Edbauer, 2013). This interactive dashboard was designed for a 
teacher in a multi-tabletop classroom. It aimed to help the teacher manage the multiple devices in the 
technology-enhanced environment and gain understanding of the students' collaboration and task 
progress. The dashboard proved effective in enhancing instructor awareness and in helping them decide 
how to direct the time and attention of students by providing informed feedback in a number of higher 
education classroom studies in classes with more than 500 students. 

3.1 Stage 1 — Problem Identification 

We describe the context of the work in terms of the problem definition as a starting point of our design 
workflow. The objective of the project is to design, evaluate, and deploy a number of visualizations of 
student group work in a classroom. The purpose of this is to enhance instructor awareness of students' 
small-group collaboration and task progress. The group activity was supported either in the lab or in the 
classroom using interactive tabletops. These shared multi-touch devices enable small groups of students 
to work on a learning task where they manipulate digital media while communicating face-to-face. 
Figure 2 (left) shows three students working on such a collaborative activity. The main requirement, 
from the LA point of view, was to generate an awareness tool (like the one shown in Figure 2, right) in 
the form of an instructor dashboard to monitor student collaboration and group progress on the 
learning activity while the students worked in small groups using multiple tabletops. 

Data sources and Data logging. Three main sources of student data were identified for the project: the 
physical manipulation of virtual objects on the tabletop, quantitative dimensions of student speech (e.g., 
presence and duration of utterances), and students' task progress (e.g., measures of the quality of the 
learning artefacts produced). Regarding the tabletop technology, there has been little research on 
sensing technology to capture the above types of student data automatically in face-to-face learning 
settings. Part of the project included the production of the technology to recognize user-differentiated 
touches on the tabletops automatically and to synchronize those with the detection of audio activity by 
the learners at the table. The sensing system CollAid was developed to capture datasets of students' 
activity at an interactive tabletop, to provide fine-grained data about learner activity similar to that 
currently collected in online learning environments. It records the timestamp of each action performed 
by each student on the multi-touch learning application. CollAid can also determine which student is 
speaking at any time. So it captures the duration of each student utterance and whether such verbal 
participation followed another student speaking. More information about how the speech and touch 
activity can be automatically captured in tabletop-based face-to-face settings can be found in Martinez- 
Maldonado, Collins, Kay, and Yacef (2011). An analysis of the accuracy of the vision-based system that 
CollAid uses to identify touches indicated that it was highly accurate for normal use by students; details 
of which can be found in Clayphan, Martinez-Maldonado, Ackad, and Kay (2013). 
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Features of the learning setting. For the prototyping stages and pilot studies, the learning activity 
involved the construction of a concept map to tackle problem solving and case-based resolution tasks 
collaboratively. Collaborative concept mapping is representative of a group learning activity that 
requires students to share their perspectives, visually represent their ideas, and agree on a group 
solution (Chaka, 2010). For the collaborative tasks at the tabletop, students used the multi-touch 
concept map editor CMate (Martinez-Maldonado, Kay, & Yacef, 2010). For the in-the-wild deployment, 
two additional learning applications were used: 1) ScriptStorm, a brainstorming application (Clayphan, 
Kay, & Weinberger, 2013); and 2) WellMet, a tool for supporting project management meetings. 
Additional details about the features of these tools can be found in Martinez-Maldonado, Clayphan, 
Ackad, and Kay (2014). 

Evaluation. In this project, classroom instructors were the main stakeholders and source of 
requirements for the tool. Therefore, their role was crucial in the evaluations discussed below. 
Additionally, a number of research goals were set for the project to explore various visual analytics tools 
and their effectiveness in conveying information about students. Thus, additional required sources were 
identified by reviewing relevant research literature. 

3.2 Series of Studies 

The design process in the case study included the following series of smaller studies, which support 
Stages 2 to 5 respectively of the LATUX workflow. The first explored a wide range of candidate 
visualizations based on paper prototypes (Martinez-Maldonado, Kay, & Yacef, 2011). The second 
consisted of trials in the lab, with teachers analyzing real student data, but simulating classroom 
sessions (Martinez-Maldonado, Yacef, Kay, & Schwendimann, 2012). The third included two pilot studies 
in real classrooms conducted in two different semesters (Martinez-Maldonado et al., 2013). The fourth 
was a longitudinal classroom experience with four instructors delivering two one-semester courses 
(Martinez-Maldonado et al., 2014). 

3.2.1 Stage 2 — Validation of prototypes 

Description of the study: The first study focused on exploring visualizations of small-group face-to-face 
work that could be generated from the data captured from a tabletop device in a lab setting such as the 
one shown in Figure 2 (left). The learning task had two stages: first, an individual stage, where each 
group member created their own concept map; and then, a group stage, where they came together to 
create a shared map using the CMate tabletop tool. The data for the study was collected for 10 
participants in three groups, each with three or four participants. Drawing on previous work on social 
proxies (Erickson et al., 1999), equity of oral participation and decision making (Bachour, Kaplan, & 
Dillenbourg, 2010), and levels of participation at the tabletop (Harris et al., 2009), four visualizations 
were generated. This initial set of visualizations was designed to provide key indicators of accountability 
of student contributions to the group solution, the progress of the task, and how egalitarian the activity 
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was amongst learners. The visualizations were generated using a computer-drawing editor. Details of 
these visualizations can be found in Martinez-Maldonado, Kay, and Yacef (2011). Two sets of the 
visualizations were printed on paper (see Figure 2, right). These sets were generated based on real data 
obtained from the video recordings of two selected groups of three students each collaborating at the 
tabletop on the concept mapping task. 

Evaluation: We illustrate a way to evaluate candidate visualizations in early stages of design using real 
student data by describing the evaluation of the paper-prototypes described above. The usability and 
meaningfulness of the visualizations were both tested using the prototypes. Regarding usability, an 
instrument was first applied to investigate if all the instructors were able to understand the 
visualizations. Then, a second instrument tested a series of hypotheses that questioned if the set of 
visualizations revealed key facets of the group collaboration processes to the instructors. These were 
about equity and amount of participation, overall collaboration, equity of intellectual contribution, and 
the creation process. In order to assess whether the visualizations conveyed enough information about 
group work, we ran evaluation sessions with five skilled instructors, each with experience in 
orchestrating classroom group work. We provided each instructor with a set of the visualizations 
generated from the tabletop sessions of two groups: A and B. We also provided the final maps, from 
both the individual and collaborative stages. Then, we invited the instructors to respond to a set of 
questions about the equity of participation, quantity of participation, the collaboration, the equity of 
intellectual contribution of the group members, and the usefulness of the visualizations to depict the 
creation map process in terms of contribution and group work. We asked the instructors to think-aloud 
while inspecting the visualizations, explain their responses, and state which visualizations they used or if 
the visualizations did not give enough support for making a decision. As a basis to compare the 
information inferred by the instructors from the visualizations, the video recordings of each group's 
session were analyzed so they could be used to compare against the instructor's thinking process. The 
instructors neither watched the videos nor had access to summaries of what actually happened during 



Figure 2. Case-study learning environment: Left: a three-member group acomplishing a 
collaborative learning task in the lab; Right: Example of a paper-based prototype for validating 
visualizations with each row showing the next visualizations: 1) the Verbal Participation Radars, 2) 
the Touch Participation radars, 3) the Map Contribution Chart, and 4) the Map Evolution Diagram. 
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the group sessions. They had just the information in the low-fidelity prototype visualizations. 

Instructors were able to use the visualizations to answer four out of the five questions. This 
demonstrated that the visualizations were promising in conveying information that could help 
instructors in monitoring some aspects of the collaborative situation at the tabletop even when the 
facilitator did not have the opportunity to observe the actions of the group in situ. This mimics real class 
contexts where the teacher devotes her attention to one group, with other groups working in parallel. 
Results showed that the visualizations provided instructors with key quantitative information about the 
equity of participation, the quantity of participation, and collaboration. However, they failed to provide 
information about the equity of intellectual contribution. Importantly, instructors indicated which 
aspects of the visualizations should be improved, which ones would be valuable for classroom use, and 
which other aspects of group work they considered important to visualize. 

Main lessons learnt: In line with the UX evaluation literature (Tohidi et al., 2006), this inexpensive paper- 
based prototyping allowed exploration of initial user experience to drive the design of the awareness 
tool. For example, the evaluation described above demonstrated how instructors can identify 
visualizations that they predicted would be useful in class and distinguished them from others that they 
would prefer to use for post-hoc reflection, outside of the demands and time pressure of the actual 
classroom. This informed the choice of goals in the following studies and the iterative design of the 
awareness tool in general. Results from this evaluation also helped to refine the design of the visual 
aspects of the group indicators as well as to detect what other visualizations would be useful to explore. 

3.2.2 Stage 3 — Simulations with real instructors 

Description of the study: Building on the results of the evaluation of low-fidelity prototypes described 
above, in this study, a high-fidelity prototype of a dashboard was built. Four instructors, all experienced 
in using collaborative classroom activities, were involved in the early stages of the dashboard prototype 
design. The design process consisted of a series of interviews, design of preliminary versions of the 
dashboards, and empirical evaluations of both the visualizations in the dashboard and the user interface 
layout. The result was a dashboard designed to enable instructors to gain awareness of group learning 
processes, to help them determine whether groups or individual learners needed attention. The 
interface showed four key facets of group work: 1) symmetry of activity, 2) degree of interaction with 
others' contributions, 3) level of collaboration, and 4) overall progress of the task. This information was 
displayed at two levels of detail: 1) the class level, which showed minimal information about multiple 
groups at the same time (with three visualizations per group); and 2) the detailed group level (based on 
Shneiderman, Borkowski, Alavi, & Norman, 1998), which allowed instructors to drill down to more 
specific information about a select group when required (with five visualizations in a timeline; see Figure 
3, right). Eight visualizations were generated and distributed on the two different user interfaces. More 
details of the study can be found in Martinez-Maldonado et al. (2012). 
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Evaluation: This study evaluated how instructors used the dashboard to intervene in a group and what 
visualizations were most effective in conveying key information to inform their decisions. Controlled 
studies were conducted in a lab with real data simulating classroom sessions. Student data captured 
from four lab sessions, with university students in groups of three, were used in the simulations. Eight 
simulated sessions were run with eight instructors (all different from the previous evaluation stage) to 
evaluate the dashboard prototype. The high-fidelity prototype simulated the real-time generation of 
data for each instructor, as if he or she was concurrently monitoring up to three groups for 30 minutes 
(as in a classroom situation). The evaluation recreated the classroom orchestration loop documented by 
Dillenbourg et al. (2011), where instructors monitor the classroom (at the class level of the dashboard), 
compare it to some desirable state, and intervene to adapt the teaching (selecting the key visualizations 
that helped in the decision making process and drilling down to the detailed group level of the 
dashboard). If the instructors decided to intervene, they had to wait at least two minutes in the detailed 
group level simulating the time taken to talk with that group. Instructors followed this loop throughout 
the duration of the trials. Data captured from the instructors' use of the dashboard, a questionnaire, and 
interviews were used to understand the instructors' experience with the tool. Additionally, similar to the 
evaluation of low-fidelity prototypes described earlier, video recordings of each group's collaboration 
was manually analyzed by an external observer. 

Evaluation results revealed that the prototype effectively enabled instructors to identify which groups 
encountered problems during the group work. This indicated that providing our key quantitative 
indicators of group work was helpful; teachers were able to decide which group needed their attention 
and to provide timely feedback. The class level of the dashboard provided information from the 
beginning of the activity and was used as a decision making tool to help teachers manage their attention 
and interventions. The detailed group level, which showed chronological information, was considered 
only to be effective for assessing task progress after class. Additionally, our participants indicated that 
those visualizations providing less processed data were more effective for the management of their 
attention and intervention. By contrast, they saw less value in the visualizations generated using more 
sophisticated data processing (a graph showing overall level of collaboration as detected by a data- 



Figure 3. Simulation of a classroom using real student data: Left: a three-member group acomplishing 
a collaborative learning task in the lab; Right: a high-fidelity prototype of an instructor dashboard to 

be used by instructors in the lab. 
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mining algorithm). 

Main lessons learnt: Even though the construction of a programmed simulator is a more expensive task, 
it provided a more realistic user experience than static paper-based prototypes (Hartson & Pyla, 2012). 
It meant that we could design a study where the instructors could experience a role-play, where they 
played a teacher who was using the visualizations and analyzing student data on the fly, as they would 
do in a real classroom. This helped them consider if they would realistically use the awareness tool in 
the classroom. For example, in this study instructors were able to identify the key features that gave 
them clues indicating groups with collaboration problems. Additionally, evaluating a prototype with two 
different levels of detail of student data provided design insights about the kind of information that 
should or should not be delivered to instructors in class. Providing more detailed information to 
instructors should be considered for cases where they want to do a careful post-mortem analysis in 
order to understand specific aspects of the collaboration or the learning processes. 

3.2.3 Stage 4 — Pilot studies 

Description of the study: The effectiveness of the implementation of the dashboard prototype in the 
classroom was validated in two consecutive semesters of authentic university level tutorials. These two 
pilot studies were conducted in two different weeks for two courses (at a Business school) with a single 
instructor conducting 22 classroom sessions (14 and 8 sessions with 236 and 140 students in each, 
respectively). Similar to the setting shown in Figure 4 (left), the classroom featured four interconnected 
interactive tabletops, each logging student actions to a central repository. All tabletops were enhanced 
with our CollAid system to identify students who touched the interactive surface, based on overhead 
depth sensors as described above. The instructor was provided with a handheld tablet, with a new 
version of the dashboard prototype (see Figure 4, right). This showed two real-time selected 
visualizations of 1) group task progress (the quality of students' work), or 2) equity of student 
participation (an indicator of group work), within each group. While the design of these visualizations 
built on lessons from the previous lab studies, we used it to create new visualizations of the information 
that the teacher wanted. 



Figure 4. Pilot studies - studies in the wild: Left: a high-fidelity prototype of an 
instructor dashboard; and Right: an instructor using an awareness tool in-class. 
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Evaluation: Learning tasks were designed by the teacher to fit into one fifty-minute weekly tutorial class 
in each course. The class sessions involved small-group discussions of a previously defined case study 
and the task required students to represent their group solutions to the posed case visually by building a 
concept map. During classes, the teacher moved freely among the tabletops, reviewing each group's 
concept map, identifying groups that needed help, and providing feedback accordingly. To evaluate the 
impact of the tool in the classroom, we collected information from a number of sources to triangulate 
evidence. These sources included application logs with user information, snapshots of the evolution of 
each group's concept map solution, the actions performed by the instructor on the orchestration tool 
and attending to groups, and notes from a second external observer whose task was to assess each 
small group's work at regular intervals. The evaluation focused on investigating if the instructor 
attended to the "lower achieving" groups, based on the information provided on the dashboard, and 
then if this intervention had some effect on student learning or on their task. Further details of this 
evaluation process can be found in Martinez-Maldonado et al. (2013) and Martinez-Maldonado (2014). 

Main lessons learnt: Results showed that the data presented to the teacher in the classroom could direct 
their attention, especially when information about the quality of student work is delivered. By contrast, 
the study also confirmed that the teacher did value indicators of group work and individual participation, 
but mostly for post-hoc analysis. This study made evident how important it is to evaluate the tools in a 
real classroom environment. In the actual class, the instructor had to cope with many constraints (e.g., 
time limits, answering student questions, aligning with the curriculum, students arriving late, organizing 
the activity, and orchestrating the whole class). In addition, our provision of the awareness tool added 
complexity to the already multifaceted instruction activity. This study also proved that an instructor 
could benefit greatly from having quick access to information that is not normally easy to access in class 
time; for example, aspects of the quality of students' evolving solutions. The realism of this study made 
it difficult to test experimental conditions (e.g., alternating visualizations in the awareness tools or 
comparing the usage of the tool with not using it at all) due to practical or ethical issues and the 
teacher's wishes (e.g., she wanted all students to have similar opportunities for learning). Additionally, 
the evaluation in these pilot studies allowed testing the impact of the actual use of the awareness tool, 
going beyond simple usability to pedagogic and learning aspects. This was not possible in the previous 
two lab studies. 

3.2.4 Stage 5 — Classroom use 

Description of the study: Finally, the awareness tool that was iteratively built and evaluated in the lab 
and in pilot studies was deployed in long-term classroom studies. This deployment consisted of a 
longitudinal class experience running across two full university (semester) courses with four instructors 
(all different from those in the previous evaluation stages). The classroom experience involved 145 
students. The multi-touch classroom ecology was used for the majority of the one-hour weekly tutorials 
for the two courses. The first was an undergraduate course on Human-Computer Interaction that had 
105 students enrolled distributed in six tutorial classes taught by three instructors. The second was a 
postgraduate course on Pervasive Computing that had 40 students enrolled divided in two tutorial 
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Figure 5. One final deployed teacher's dashboard featuring A) functions to control the tabletops; B) 
functions to control the vertical 'wall' displays; C) small group visualizations; and D) notifications 


classes taught by two tutors. In this study, a full version of the awareness tool was provided. Figure 5 
shows an example screen. The interface provided functions for the instructor to control the tabletops 
and vertical displays in the classroom (Figure 5-A and 5-B). For example, to block all the tabletops so the 
instructor could give instructions to the students, to move all the tabletops to a different phase of the 
class task, or to send the content of a specific tabletop to be displayed on a classroom vertical "wall" 
display. It also showed one visualization per group (Figure 5-C) and it included the provision of 
notifications (squares with rounded corners around visualizations — Figure 5-D) that were automatically 
triggered when misconceptions were detected or when groups achieved at least 50% of the task as the 
instructor had pre-defined it. The misconceptions were detected by comparing the runtime of each 
group's concept map with a list of common misconceptions defined by the teacher in design time. The 
dashboard reflected the change in student behaviour or the interventions performed by the instructor. 
For example, if the instructor attended to a group that had a misconception, the dashboard updated in 
runtime when (and if) the group had addressed that misconception. 


Evaluation: Similar to the previous study, testing the awareness tool in-the-wild provided understanding 
of the impact of the tool on learning and instruction. However, a full semester deployment shed light on 
additional requirements for a truly useful teacher's dashboard. First, each week, the classroom sessions 
were different and instructors had to follow a particular script. Then, the notion of a script and a 
visualization portraying its enactment in real time were introduced. The group visualizations had to be 
generated in this context, matching the stage of the script where the class was up to, at any given time. 
Moreover, it was important for instructors to switch the active learning applications on the tabletops 
easily. Thus, the visualizations had to convey meaningful indicators of group work for different types of 
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tasks supported by three different tabletop applications (in our case: concept mapping, brainstorming, 
and group meetings). Each of these learning applications was used at least twice during the semester in 
each course. Results showed how the different instructors used the visualizations according to their 
previous teaching experience, personal styles and each particular task each week during the courses. In 
some weeks the dashboard was more crucial, especially in those tasks where particular teaching goals 
had to be accomplished (e.g., when equal contributions from each group member were expected or 
when instructors had to revise knowledge of particular topics). By contrast, for more open tasks such as 
team meetings, the workload of regulating the class sessions was more distributed among students; 
hence, the instructors did not have to monitor task progress or group achievement explicitly. 

Main lessons learnt: The evaluation of the awareness tool in-the-wild in this larger study gave insights 
about a higher degree of generalization of the results (Brown, Reeves, & Sherwood, 2011). As in any 
design task, the visualizations in the dashboard are not necessarily final. However, the LATUX processes 
demonstrated that they are useful and meaningful. This held true for more than just the concept 
mapping activity, to include other learning activities as well (brainstorming and face-to-face meetings 
and coordination of a major multi-week project). Additionally, the in-the-wild study allowed for testing 
the tool with more instructors than the pilot studies and for a longer time, thus minimizing the novelty 
effect. 


3.2.5 Iterative Design 

The design of effective awareness tools, similar to other user interfaces, calls for an iterative process. 
The purpose of this case study is not to recommend all the details of these visualizations and 
dashboards themselves. For this there has been nascent but extensive work (Verbert et al., 2013). 
Rather, we used it to illustrate the process of designing and refining visual awareness tools that may be 
effective for a specific context. For example. Figure 6 shows the evolution of two visualizations that 


Paper prototype Simulation prototype 


In the classroom 



Contribution 

Charts 


Links: 14 
Concepts: 15 


Concepts:31 Links:27 16 links 



Figure 6. Evolution of two visualizations, from prototypes to deployment 
A) Radars of Participation and B) Contribution charts. 
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were deployed in a classroom and were initially evaluated with prototypes. Although simple, they 
provide useful information to the instructor. It can be observed that: 

1. There was a tendency towards minimalism. For example, for the visualizations "radars of 
participation" (see the three visualizations shown in Figure 6, top row) the first prototypes had 
two triangles depicting equity of verbal (blue) and touch (red) participation with the interactive 
tabletops. In the classroom, given the challenges for capturing clean speech data, the 
visualization had to be simplified to show just touch activity data from the tabletops and names 
of students rather than symbolic representations (coloured circles). This observation is partly in 
line with the data-ink ratio principle (Tufte, 1983). This principle promotes the avoidance of 
distracting elements as part of the visualizations design, which could be observed as the two 
visualizations evolved in each iteration. The principle also promotes the non-redundant display 
of data-information, which could be observed in the evolution of the radars of participation. 
However, some redundancy remained in the last design of the contribution charts (e.g., the 
number of links represented both as a number and as a coloured area). 

2. Additionally, there was an increased focus on higher-level indicators and less on accountability. 
For example, for the "contribution charts" (see the three visualizations shown in Figure 6, 
bottom row), the pie chart used for the prototypes was simplified to indicate the size of the 
solution (outer orange circle, in the rightmost visualization) and the portion that matched 
essential elements expected by the instructor (the inner yellow circle) instead of individual 
contributions depicted by the slices of the pie chart. 

This project is continuing to explore other visualizations of group activity that can be useful to 
instructors or students. Therefore, it is important to have an iterative perspective from the beginning of 
the design process though to the end. 

4 CONCLUSIONS AND RESEARCH AGENDA 

The design of effective LA tools requires the convergence of methodologies from multiple disciplines 
such as software engineering, human-computer interaction, and education. Even though these 
disciplines provide guidelines and frameworks to guide designers towards effective designs and systems, 
the LA community may benefit from paradigms that reflect its additional and specific multidisciplinary 
needs. There is a need for systematic processes to tackle the critical challenge of designing effective 
interfaces for LA applications. Our work set out to tackle the recognized need for a principled approach 
for designing the interfaces of LA tools. In this paper, we have reported the tabletop case studies to 
illustrate how to use the LATUX workflow. The series of case studies also demonstrate that the process 
resulted in valuable LA interfaces for learning and teaching with interactive tabletops. Figure 1 provided 
an overview of the five stages of the workflow, with questions for the LA tool designer to consider. The 
LATUX workflow calls for an iterative process, with five stages that build towards the objective of 
producing robust tools suitable for large-scale adoption, and based on the combination of well- 
established techniques to improve user experience while maintaining an explicit connection with the 
underpinning learning environment. Our LATUX workflow draws on the substantial body of work from 
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SE and UX disciplines but also considers the pedagogical requirements and challenges of designing LA 
tools to support teaching and learning in technology-enabled learning environments. We now discuss 
the key lessons for using the LATUX workflow. 

The reader may be concerned about the amount of effort involved in our series of studies to provide LA 
tools for tabletop classrooms. However, this is largely due the significant effort needed to work with 
very untried learning technology. We wanted to harness the digital footprints from students' touch of 
the tabletops and the recorded audio of their conversations. This meant that we tackled the process 
from a very fundamental starting point: the development of the data collection technology. For many 
valuable uses of LA, there are established, well understood data sources. For such data sources (e.g., 
data captured from the learning management system or video technologies), we expect that the 
progress from low-fidelity to high-fidelity prototypes and on to pilot and classroom studies can be done 
with modest effort. As we assemble more case studies of evaluated LA tools, we can expect to create 
ever-more successful LA interfaces with fewer iterations. 

In our work, a key challenge was to consider the LA goals at the same time as the design of the learning 
activities. This was valuable, as it demanded careful thought, decision-making, and documentation of 
the learning objectives. Initially, we set out to provide LA tools for visualizing group collaboration. When 
we moved to the classroom, the topic-level learning outcomes had to be incorporated. In many widely 
used learning platforms, we still need to build a larger set of case studies for the Problem Identification 
stage. Notably, it is likely to be valuable to improve the primitives for linking learning outcomes to the 
available data. We then need case studies that move through the other four stages. The LATUX 
workflow has a role to play in supporting this exploration. In the long term, we expect that a set of core 
classes of requirements will emerge. For example, many online learning tools could usefully provide LA 
tools that show student progress in the set learning objectives. This is similar to our tool showing 
progress in creating the teacher's target elements in the concept maps, as well as anticipated 
misconceptions. 

We emphasize that our contribution is the process. In addition, the series of case studies discussed 
represent a valuable foundation for creating tools for teachers to use for classrooms that involved small 
group learning activities where data is available about individual contributions and about the progress of 
the group task. These findings are likely to be useful for other contexts that share the same goal of 
enabling a teacher to see the progress of multiple small groups and to decide how to direct her 
attention to groups that most require it. This may also be useful for online collaborative learning 
activities, with multiple small groups and set class times. 

The LATUX workflow has been supported with a series of case studies on the development of an 
awareness tool for instructors to observe small group interactions in a classroom. Even though the 
studies discussed involve rather leading-edge technology (e.g., interactive tabletops), the core lessons 
learnt, and the proposed workflow have broader applicability to both existing and future learning 
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technology. Further work is needed to validate the application of the workflow in blended and online 
learning environments, and explore cases where awareness tools are given to students or other 
stakeholders. Future work is also needed on aligning the LA solutions with the design of the learning 
activities (including the configuration of tools, tasks, and pedagogical strategies) to support instructors 
more effectively. We believe that this work is an initial step towards much needed research to provide 
methodologies for the design of LA awareness tools. We consider the area of design frameworks as a 
crucial aspect to contribute to the holistic view of LA, foster its widespread adoption, and improve the 
overall learning and teaching experience. 
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